{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Multi Layer Perceptron\n",
    "=============\n",
    "\n",
    "The Multi Layer Perceptron (MLP) is an extension of the classical [Perceptron](https://en.wikipedia.org/wiki/Perceptron) having one or more hiddel layers. Historically we must remember that after the [book of Minsky and Papert](https://en.wikipedia.org/wiki/Perceptrons_(book) in 1969 the research on neural networks was abandoned. In 1985 Rumelhart, Hinton and Williams published an [article](http://www.dtic.mil/docs/citations/ADA164453) on the use of a generalized delta rule for training an MLP. Rumelhart et al. experimentally verified that using the additional layer and the new update rule the network was able to solve the XOR problem. This result ignited again the research on neural networks. The authors stated:\n",
    "\n",
    "*“In short, we believe that we have answered Minsky and Papert's challenge and have found a learning result sufficiently powerful to demonstrate that their pessimism about learning in multilayer machines was misplaced.”*\n",
    "\n",
    "The MLP in its classical form, is based on an input layer, an hidden layer and an output layer. The transfer function used between the layers is generally a Sigmoid. The loss function can be defined as the mean squared error between the output and the labels. Each layer of the MLP can be represetned as a vector-matrix multiplication between an input vector $\\boldsymbol{x}$ and a weight matrix $\\boldsymbol{W}$. The resulting value is added to a bias and passed to an activation function, generating an output vector $\\boldsymbol{y}$. These operations are equivalent to the weighted sum of the input values.\n",
    "\n",
    "<p align=\"center\">\n",
    "<img src=\"../etc/img/mlp_model.png\" width=\"500\">\n",
    "</p>\n",
    "\n",
    "It is possible to stack multiple hidden layers into a single feedforward network. An MLP having multiple hidden layers can be defined as a **deep** neural network. However, deep MLP are not so common because the dense connections introduce many parameters and in very deep models this cause an explosion in the computational resources needed to manage it. However, the MLP is often used as the last stage of deep convolutional neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implementing the model in Tensorflow\n",
    "------------------------------------------\n",
    "\n",
    "It is straightforward to implement the model in Tensorflow. Using the `tf.layers` facilities we can define a perceptron in three lines of code. Here I will use the implementation based on the `Estimator` class that requires to embedd the model into a function and associate it to the estimator object. The model is automatically stored in a folder (specified when you create the estimator) and a checkpoint is saved during the training. Thanks to the checkpoint you can resume the training at any time. The output value returned by the MLP is a real number between 0 and 1 given by the sigmoid. The output can be approximated to the closest integer using `tf.round()`. Through the round off it is possible to interpret the results in term of classification and print the accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_model_fn(features, labels, mode):\n",
    "    #Defining the MLP model\n",
    "    x = tf.reshape(features, [-1, 2])\n",
    "    h = tf.layers.dense(inputs=x, units=8, activation=tf.nn.sigmoid)   \n",
    "    y = tf.layers.dense(inputs=h, units=1, activation=tf.nn.sigmoid)\n",
    "    #PREDICT mode\n",
    "    if mode == tf.estimator.ModeKeys.PREDICT:\n",
    "        predictions = {\"classes\": tf.round(y),\n",
    "                       \"probabilities\": y}\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
    "    #TRAIN mode\n",
    "    elif mode == tf.estimator.ModeKeys.TRAIN:\n",
    "        loss = tf.losses.mean_squared_error(labels=labels, predictions=y)\n",
    "        #optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)\n",
    "        optimizer = tf.train.AdamOptimizer()\n",
    "        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())\n",
    "        accuracy = tf.metrics.accuracy(labels=labels, predictions=tf.round(y))\n",
    "        tf.summary.scalar('accuracy', accuracy[1]) #<-- accuracy[1] to grab the value\n",
    "        logging_hook = tf.train.LoggingTensorHook({\"accuracy\" : accuracy[1]}, every_n_iter=250)\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op, training_hooks =[logging_hook])\n",
    "    #EVAL mode\n",
    "    elif mode == tf.estimator.ModeKeys.EVAL:\n",
    "        loss = tf.losses.mean_squared_error(labels=labels, predictions=y)\n",
    "        accuracy = tf.metrics.accuracy(labels=labels, predictions=tf.round(y))\n",
    "        eval_metric = {\"accuracy\": accuracy}\n",
    "        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = tf.estimator.Estimator(model_fn=my_model_fn, model_dir=\"./tf_mlp_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the model\n",
    "---------------------\n",
    "\n",
    "Once we have the model ready, we can train it on a dataset. Here I will use the **XOR dataset** that has been created in [another notebook](../xor/xor.ipynb) of this repository. You do not have to run the notebook, since a version of the dataset has been included in TensorBag and is ready to be used. With the estimator class of Tensorflow it is necessary to pass an input function to the trainer. Here I define this function and I parse the dataset that is stored in TFRecord format. The dataset is allocated as a Tensorflow `Dataset` object, that makes very easy to return samples from it. Remember that you can monitor the training using **Tensorboard** through the `--logdir` parameter from the terminal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_input_fn():  \n",
    "    def _parse_function(example_proto):\n",
    "        features = {\"feature\": tf.VarLenFeature(tf.float32),\n",
    "                    \"label\": tf.FixedLenFeature((), tf.int64, default_value=0)}\n",
    "        parsed_features = tf.parse_single_example(example_proto, features)\n",
    "        feature = tf.cast(parsed_features[\"feature\"], tf.float32)\n",
    "        feature = tf.sparse_tensor_to_dense(feature, default_value=0)\n",
    "        label = tf.reshape(parsed_features[\"label\"], [1])\n",
    "        label = tf.cast(label, tf.float32)\n",
    "        return feature, label\n",
    "\n",
    "    tf_train_dataset = tf.data.TFRecordDataset(\"../xor/xor_train.tfrecord\")\n",
    "    tf_train_dataset = tf_train_dataset.map(_parse_function)\n",
    "    tf_train_dataset.cache() # caches entire dataset\n",
    "    #Setting a buffer_size greater than the number of examples in the Dataset \n",
    "    #ensures that the data is completely shuffled. \n",
    "    tf_train_dataset = tf_train_dataset.shuffle(buffer_size = 8000 * 2) # shuffle all the elements\n",
    "    #The repeat method has the Dataset restart when it reaches the end.\n",
    "    tf_train_dataset = tf_train_dataset.repeat() # repeats dataset this times\n",
    "    #The batch method collects a number of examples and stacks them, to create batches. \n",
    "    #This adds a dimension to their shape. The new dimension is added as the first dimension.\n",
    "    #The batch may have unknown batch size because the last batch can have fewer elements.\n",
    "    tf_train_dataset = tf_train_dataset.batch(32) # batch size\n",
    "    \n",
    "    iterator = tf_train_dataset.make_one_shot_iterator()\n",
    "    batch_features, batch_labels = iterator.get_next()\n",
    "    return batch_features, batch_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.logging.set_verbosity(tf.logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mlp.train(input_fn=my_input_fn, steps=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluation on the test set\n",
    "------------------------------\n",
    "\n",
    "The XOR dataset also has a test set that can be used to estimate the accuracy of the MLP. Here we use the `eval()` method of the Tensorflow Estimator class and we pass an input function that pre-process the dataset. The result is in term of loss and accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_eval_input_fn():\n",
    "    def _parse_function(example_proto):\n",
    "        features = {\"feature\": tf.VarLenFeature(tf.float32),\n",
    "                    \"label\": tf.FixedLenFeature((), tf.int64, default_value=0)}\n",
    "        parsed_features = tf.parse_single_example(example_proto, features)\n",
    "        feature = tf.cast(parsed_features[\"feature\"], tf.float32)\n",
    "        feature = tf.sparse_tensor_to_dense(feature, default_value=0)\n",
    "        label = tf.reshape(parsed_features[\"label\"], [1])\n",
    "        label = tf.cast(label, tf.float32)\n",
    "        return feature, label\n",
    "\n",
    "    tf_test_dataset = tf.data.TFRecordDataset(\"../xor/xor_test.tfrecord\")\n",
    "    tf_test_dataset = tf_test_dataset.map(_parse_function)\n",
    "    tf_test_dataset.cache() # caches entire dataset\n",
    "    tf_test_dataset = tf_test_dataset.repeat(1) # repeats dataset this times\n",
    "    tf_test_dataset = tf_test_dataset.batch(1) # batch size  \n",
    "    \n",
    "    iterator_test = tf_test_dataset.make_one_shot_iterator()\n",
    "    batch_features, batch_labels = iterator_test.get_next()\n",
    "    return batch_features, batch_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mlp.evaluate(input_fn=my_eval_input_fn, steps=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the model on custom data\n",
    "----------------------------------\n",
    "\n",
    "To use the model on custom data it is possible to use the `predict()` method of the estimator class. Also in this case an input function must be used in order to return the input features. You may need the truth table of the XOR operator:\n",
    "\n",
    "- 1 XOR 1 = 0\n",
    "- 1 XOR 0 = 1\n",
    "- 0 XOR 1 = 1\n",
    "- 0 XOR 0 = 0\n",
    "\n",
    "When the input values are equal (same sign) the output is False (zero). When the input values are different, the output is True (one). Here I hardcoded some values that belong to the four quadrants of the XOR plane. The output is printed on the terminal and should be: zero, one, one, zero. You can also try different values and verify the output from the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def my_predict_input_fn():\n",
    "    feture_batch = tf.constant([[3.5, 2.9], [3.5, -2.9], [-3.5, 2.9], [-3.5, -2.9]])\n",
    "    \n",
    "    tf_predict_dataset = tf.data.Dataset.from_tensor_slices((feture_batch))\n",
    "    tf_predict_dataset = tf_predict_dataset.repeat(1)\n",
    "    \n",
    "    iterator_predict = tf_predict_dataset.make_one_shot_iterator()\n",
    "    batch_features = iterator_predict.get_next()\n",
    "    return batch_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predictions = mlp.predict(input_fn=my_predict_input_fn)\n",
    "\n",
    "for i, prediction in enumerate(predictions):\n",
    "    print \"Predicted class: \" + str(prediction['classes'])\n",
    "    print \"Probabilities: \" + str(prediction['probabilities'])\n",
    "    print \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Improving the performance\n",
    "-----------------------------\n",
    "\n",
    "You can further improve the performance changing the **structure of the network**. For instance, you can increase the number of hidden units. You must be careful because when the number of units is too large you may encounter [overfitting](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) problems. Another important factor is the **learning rate**. In this tutorial we used Adam as adaptive gradient method for setting the learning rate. It has been shown in the [article](https://arxiv.org/pdf/1412.6980.pdf) of Kignma and Lei Ba (2015) that Adam perform better than other methods, leading to a lower loss in the same amount of iterations. Comparing different adaptive gradient methods on the MNIST dataset, shows that Adam has an optimal performance.\n",
    "\n",
    "<p align=\"center\">\n",
    "<img src=\"../etc/img/mlp_optimizers.png\" width=\"300\">\n",
    "</p>\n",
    "\n",
    "Altough the Adam optimizer seems to be the best choice in many classification problmes, you can also try different optimizers. For instance, you can train an MLP on the MNIST dataset and verify the results of Kignma and Lei Ba (2015) on different adaptive methods.\n",
    "\n",
    "Conclusions\n",
    "-------------\n",
    "\n",
    "In this tutorial we saw how the MLP works and how it is possible to train it on a simple non-linear dataset. A good exercise is to train the MLP on a different dataset. A valid dataset is the [Iris Flower](../iris/iris.ipynb), available on this repository. To train the MLP on a new dataset you may need to change the number of units in the input and output layers, and the loss function. You can follow the [Quiz notebook](./mlp_quiz.ipynb) to implement an MLP for the Iris Flowers dataset. Good luck!\n",
    "\n",
    "**Copyright (c)** 2018 Massimiliano Patacchiola, MIT License"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
